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The Dunford House Seminar 



This is an annual residential seminar run by the 
English Language and Literature Division of the British 
Council as part of its commitment to the provision of 
training and updating for ELT specialists. The seminar 
serves not only British Council career officers but also 
Council-recruited ELT staff - those working on schemes 
funded by the Overseas Development Administration 
and those employed in the Council's own language 
centres. It also, for the first time this year, included 
participants irom public and private sector institutions. 
The desirability of promoting exchange of experience 
between these various groups guides the selection of 
the participants. During the two weeks spent at 
Dunford House in West Sussex the participants are able 
to meet and discuss issues with leading British 
academics and ELT Council staff in similar situations to 
themselves from all over the world, and to exchange 
ideas and experiences in both formal and informal 
settings. 

Previous seminar topics 

1978 ESP course design 

1979 ELT course design 

1980 Communicative methodology 

1981 Design, evaluation and testing in English 
language projects 

1982 Teacher training and thr curriculum 

1983 Design and implementation of teacher training 
programmes 

1984 Curriculum and syllabus design in ELT 

1985 Communication skills training in bilateral aid 
projects 

1986 Appropriate methodology 

1987 ELT and development: the place of English 
language teaching in aid programmes 

Copies of reports of these seminars are available on 
request from: 

English Language and Literature Division 
The British Council 
10 Spring Gardens 
London SW1A 2BN 
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Editorial note 



This report on the 1988 Dunford f>eminar has been 
deliberately produced in a simpliiied format, since it 
was felt that the full details of discussions and 
case-study outputs would not be of interest to readers 
who were not themselves present throughout the 
seminar. A full record of the procedures and products 
of the seminar is held both in English Language and 
Literature Division at the British Council and in 
Education Division at the ODA where those interested 
may consult it. 
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Foreword 



Dunford House: value for money 

I little thought, some twenty years ago when I walked 
through an attractive bluebell wood near the house of a 
friend of mine in Sussex, that two decades later I 
shculd find myself staying in the beautiful house that 
stood beyond the faint blue mist of flowers through the 
trees. Dunford is an ideal setting for a group of 
refugees from the EFL battlefront! 

This year saw a far closer ODA involvement in the 
planning of the seminar and a sharper focus on the 
management of our ELT programmes overseas. This 
arose both from a general sen. r « in ODA that past 
seminars had involved participants in looking too 
narrowly at purely academic issues rather than at the 
broader context of their work overseas. ODA funds 
ELT activity because it believes that this is one means 
of furthering social and economic development, and any 
desk officer in ODA is keenly aware that the money 
spent on ELT is not money spent on some other 
essential purpose, such as hospital equipment, rural 
development, food production, or housing for the poor. 

We were therefore looking to Dunford to provide a 
comprehensive setting for ELT activity. We wanted to 
place ELT within an economic context, to examine how it 
contributes to overall manpower development in the 
recipient country, and how it is organized and 
managed. As in the previous year, the project 
framework was the starting-point, since this provides a 
basis for looking at the wider purposes behind any 
particular project. 

It was encouraging, on my two visits, to find such a 
lively and well-motivated group of people talking with 
enthusiasm about their overseas work. It was 
especially encouraging to find field-staff more acutely 
aware of ODA thinking than most equivalent ranks in 
other disciplines. It was encouraging to see how far 
those prese.it were moving towards an understanding of 
the broader purposes of using scarce developmer f , 
money on their particular project. 

Nevertheless, I think that we have still to focus the 
meeting even more sharply. I am far from convinced 
that the many visiting speakers who attend Dunford 
always provide the kind of contribution that will 
develop the central theme; there is a tendency for 
outsiders to be brought in to talk only in broad general 
terms about ELT. Nor, indeed, do I believe that the 
seminar should focus entirely on ELT; since the 
teaching of English in the overseas development context 
is not an end in itself but a means of providing a medium 
for acquiring other skills, it would make better sense 
for the seminar to include participants from the key 
disciplines which English skills are supposed to serve: 
science and mathematics, for example. I am sure that 
the presence of two non-ELT participants this year 
helped to move everyone's gaze upwards and outwards 
to wider issues. 



This is an important point. The money that pays for 
Dunford comes from the Education Development Awards 
fund. It competes with the money that is spent on 
providing study opportunities for field-staff in need of 
higher academic qualifications. Every pound spent on 
Dunford House is a pound that cannot be used for a 
recipient of an EDA, and a careful balance therefore 
must be struck between the needs of EDA and the 
length and scope of the Dunford Seminar. 

For the time being we both appreciate and can afford 
the Dunford Seminar, but we should continue to 
scrutinize its length and focus. Then we may be able 
to increase its effectiveness in orienting field-staff 
towards their role at the cutting edge of human 
resources development in countries where the poor have 
many additional needs. 

Dr R O Iredale 

Chief Education Adviser, ODA 



The seminar generally was also too diffuse. Two weeks 
is far too long for the kind of intensive experience that 
most participants need or want. More intensive 
sessions, less time spent on the important but 
non-specific issues of recruitment and job advertising, 
and fewer focused external sessions would all help to 
reduce the seminar's length, making it both more 
intensive and cheaper. 
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Introduction 



To be associated with the Dunford Seminar is always a 
pleasure and as the reputation of the seminar has grown 
over the years it has become a privilege as well. It is 
particularly gratifying to be writing an introductory 
piece for the report on the 1988 seminar and, since I 
retire next year, I feel I can allow myself both to reflect 
on seminars past with which I have been closely 
associated, and to speculate a little on the future. 

Like all worthwhile ventures, the Dunford Seminar has 
changed and developed over the years in response to the 
changing needs and interests of those for whom it is 
organized. Prom tne very beginning in 1978 it h,*s been 
a forum for the discussion of ideas and developments 
which have been at the leading edge of our profession 
of ELT in the overseas context, both as practitioners 
ourselves and increasingly as advisers and trainers. I 
believe that it has been through maintaining the two 
principles of quality and relevance as essential features 
of its planning that the seminar has deservedly earned 
its standing not only among those of us who carry out 
our work overseas, as KELT Advisers or Council 
officers, but also among those at the forefront of ELT 
in Britain, many of whom have given generously of 
their time to speak at the seminar over the years. 

The 1988 seminar and that of 1987 are clear illustrations 
of the way in which the theme of the seminar can change 
to meet the needs of the moment. Both have focused 
upon the managerial aspects of Britain's aid programme 
in the field of ELT, and the effects that projectisation on 
the one hand and the integration of ELT Aid within the 
country-specific policies of ODA geographical desks on 
the other, have had and will continue to have on the 
work of all those involved in the field. The 1988 
seminar has been notable for the close consultation 
which took place with the ODA at the planning stage 
and the greater involvement of ODA staff during the 
seminar itself and I am sure that such close 
collaboration was valuable for all concerned. 

A further example of a new development was the 
full-time participation for the first time of two people 
from outside the field of ELT, both of whom provided 
important contributions to the work of the seminar as a 
whole. Francis Lopez-Real, a mathematics education 
expert currently serving in Cameroon, provided vital 
information and a new dimension to the discussions and 
work of the case-study group who were looking at the 
integrated project in Cameroon involving aid to 
secondary-level teaching of mathematics, physics and 
English. Mike Ratcliffe, who is currently serving as 
ODA's Education Adviser in the Caribbean, brought 
with him his knowledge of how ELT fits in to the broader 
picture of educational aid in that region as well as his 
own science education background. 

The Dunford Seminar has become over the last few years 
the major, indeed the only, regular gathering of British 
expertise in the field of ELT in the developing world. 
It provides an important opportunity for KELT Advisers 
to meet each other and exchange views, opinions, and 
experiences. This function of the seminar is, in my 
view, very important, since by the nature of their jobs 
KELT Advisers are professionally scattered and isolated 
around the world and there are few avenues for them to 
discuss their professional problems and solutions with 
each other. I sinctrely hope that the Dunford Seminars 
of the future will continue to fulful this need. 



Dunford has always had a training function to a certain 
extent but training has never been the sole objective of 
the seminar. A parallel function has been to act as a 
developmental event for the profession, not just for the 
individual, and to be in a sense a think-tank for those 
working in the field. It seems to me that it is essential 
for the continuing growth of British expertise and its 
application in so many contexts around the world that 
this opportunity should continue for us to reconsider 
existing practice and break new ground. 

ET J Phillips 
Controller 

English Language and Literature Division 
The British Council 



Project evaluation: concerns and challenges 



Introduction 

The Dunford Seminar of 1988, held once again at 
Dunford House, was a logical successor to the 1987 
reminar in that it continued to focus on Britain's aid 
programme in the field of ELT and on the effects of the 
shift towards more tightly planned and focused 
projects, integrated in policy terms within the overall 
aid programme for each particular country. The 1987 
seminar had concentrated on examining the primary tool 
for projectisation, the Project Framework, and seeing 
how it could be applied to projects in the field of 
education in general and ELT in particular. The 
primary focus in 1988 was on the monitoring and 
evaluation of projects set up within such frameworks. 

The background to both seminars is the climate of 
thought that pervades all areas of government at the 
moment. 'Value for money' is the watchword. The 
ODA, like every ether government department and 
agency, is under pressure from the Public Accounts 
Committee and from the National Audit Office to 
demonstrate that the funds at its disposal are 
effectively spent; this pressure is transmitted down 
the line to the British Council as a major executive 
agent of the ODA and thence to the practitioners in the 
field - KELT Advisers, Technical Co-operation Officers 
(TCOs) and all those who work in ODA-financed 
projects. 

Value for money 

The concept of Value for money* implies a number of 
things, all of which have further implications running 
down to the day-to-day implementation of projects. What 
are these considerations and implications? What are the 
practical effects they have upon the way in which 
projects are implemented? How can they be measured? 
Are the measures proposed consonant with the 
objectives of educational aid or in conflict? Do the 
measures provide an adequate picture or a distorted or 
partial one? What are the constraints which may affect 
the way in which we evaluate projects? These are the 
questions which underlay all the discussions during the 
seminar. 

There are three headings or value -types under which 
one might seek to define the value of a bilateral aid 
programme and the choice will significantly affect the 
way in which any project is assessed. The three types 
of value might be labelled 'political', developmental' 
and cultural' and correspond roughly to the interests 
and objectives of the three bodies involved in the kind 
of projects under discussion at the seminar, namely the 
Foreign Office, the ODA, and the British Council. The 
three headings are not watertight compartments and all 
projects in education relate in varying degrees to all of 
them, but it is important to understand clearly which of 
the three is the primary one in the project or 
programme under consideration. 

In both the opening and closing sessions of the seminar 
Dr Roger Iredale, ODA's Chief Education Adviser, 
stressed that for the ODA it is the developmental 
Value-type that is pre-eminent and emphasized that 
projects must be planned in relation to the contribution 
they make to the social and economic development of a 
country, with a focus on the alleviation of poverty. It 
is against this criterion that they will be assessed. It 



was stressed throughout the seminar that project 
frameworks should make explicit reference to the 
economic and social benefits to be derived from the 
project in the section dealing with 'wider objectives' 
and that the indicators of achievement should be 
measurable in economic terms and in relation to the 
workforce requirements of the country. 

Moreover, this 'value' must not be just demonstrable 
but measurable, since without measurement the 
relationship of value-of-output to cost-of-input cannot 
be demonstrated. 

Concerns 

Assessing Value tor money' involves a number of 
criteria against which a project can be measured. 
These include management efficiency, which demands 
the existence of clear objectives and proper financial 
control, and cost-effectiveness, which involves the 
search for and establishment of simpler implementation 
procedures and performance indicators. Brian Vale, 
Assistant Director-General of the British Council, spoke 
of the way in which the ODA and the British Council 
were reviewing current schemes and their operational 
procedures, while Jack Phillips, Controller, English 
Language and Literature Division, summarized the 
response of his division to the new situation, including 
the creation of an ELT Aid Unit with responsibility for 
developing and evaluating more cost-effective 
procedures for the management of ELT projects on 
ODA's behalf. Roger Iredale spoke of the Project 
Framework as a 'splendid tool' for ensuring that 
projects are designed with clearly specified objectives 
with necessary inputs detailed on a time-scale, and with 
clear job descriptions for project personnel. 

Monitoring throughout the lifetime of projects is an 
essential feature of the project approach. At many 
levels of project implementation, such as staff 
recruitment or equipment procurement, measures 
already exist or can be devised with relative ease to 
assess performance, and in many kinds of aid project it 
is not difficult to devise adequate quantifiable measures 
for all levels of implementation. But there are a number 
of features of educational projects which make them 
difficult subjects for evaluation. 

Four of these features were in the forefront of 
participants' minds throughout the seminar and merit 
further discussion here. 

The starting-point 

First is the fact that, in most, if not all educational 
projects, we are involved in seeking to develop or 
change not only products (the number of graduates in 
English, for example), but the processes and systems 
which produce the products. It is here that the 
difficulty lies, since to measure change we need to know 
where we start from and it is very difficult to establish 
with any degree of exactitude where we are in the field 
of education. If, for example, a project aims to change 
the way in which teachers teach, we need to describe 
how they teach before the project begins. This is far 
from easy given the numbers involved, the relative 
autonomy of teachers in their class-rooms, and the 
danger that the process of observation may change what 
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is being observed - an observed lesson is not 
necessarily a normal lesson. 

It is in relation to this problem that the ODA is 
interested in developing techniques for carrying out 
effective baseline studies for education projects. Good 
practitioners in the field have tended to do this only in 
a non-explicit, informal way and this makes it difficult 
for them to demonstrate objectively to the outsider or 
the eventual evaluator that change has indeed taken 
place. The need therefore is to develop baseline 
studies which are explicit and which result in 
information that is comprehensible and acceptable not 
only to the specialist practitioner in the project but also 
to the non-specialist outsider. Free-standing baseline 
studies carried out by independent consultants are too 
time-consuming and expensive to become the norm, 
though it is essential that some be carried out in this 
way in order to develop techniques. It may well be that 
in the future many projects will begin with an explicitly 
exploratory phase leading to a more careful and detailed 
definition of objectives than is the case at present. 

The time-scale 

Second is the fact that educational change is a long and 
slow process. The real effects of a project may not be 
apparent until long after the project itself has ended in 
the sense of financial commitment by the ODA. This 
difficulty is compounded by the financial rules within 
which the ODA operates, since it is not allowed to make 
any binding commitment for more than three years into 
the future. This constraint upon the long-term view 
increases the pressure to fund projects which can be 
expected to produce measurable results in the shorter 
term and conflicts with the realities of projects in 
education. A partial solution to the problem is the 
identification of phases within an educational project, 
each of which can have its own, intermediate, objectives 
which produce measurable results enabling funding to 
be secured for a subsequent phase. 

The question of time-scale was brought out by Brian 
Rockcliffe in his exposition of the approach taken by 
VSO to projects in the field of education. VSO was 
very conscious of the need to take a long view and was 
fortunate not to be bound by the constraints of 
Treasury financial regulations in this respect. As an 
example of an aid agency taking a really long-term view 
he cited the case of the West Germans, who had made a 
commitment to one project in Sierra Leone for at least 
twenty years. 

Moreover, the variability introduced by the human 
element means that the end results even when 
developmentally relevant and significant, may not be 
those foreseen at the outset of the project. There is 
thus a double danger - that of making a judgement on 
the success or failure of a project too soon, before the 
real effects have manifested themselves, and that of 
making a judgement after measuring things which, while 
apparently important and relevant at the inception of 
the project, are no longer appropriate. It is in relation 
to this problem that the role of monitoring becomes 
crucial. Only an effective set of monitoring procedures 
can enable us to keep track of the rate of progress of 
the project towards its objectives and make an informed 
estimate of when it can be expected to reach them, and 



can indicate whether the project activity is leading 
directly to those objectives or whether there is some 
'drift' away from the target, or show that the target 
itself has shifted. Without some such record of project 
development or change we lose the ability to evaluate it 
in any real sense. 

The human factor 

The third major feature of educational aid projects is 
that they of necessity and almost by definition involve 
people in an intimate way that is not true of most other 
kinds of aid. Educational aid projects cannot succeed if 
they do not have the active and positive participation, 
involvement and commitment of the recipients. It is not 
pur own system or our own behaviour that we are 
seeking to change, and it is only those who live, work 
and belong within the system who can bring about any 
long-term change and then only in such measure as 
they are able and willing to do. This has major 
implications. First, if commitment is to be fostered 
there must be local involvement in the on-going 
monitoring of the project and this means that the 
criteria for monitoring and evaluation cannot solely be 
those of the donor. While in many fields of aid the 
criteria for evaluation will be the same for both parties, 
in education, dealing as it does with people and 
cultures, this may be more problematic. 

Second, it becomes very important that monitoring and 
evaluation be not solely summative and judgemental but 
that it be as illuminative as possible so as to help in the 
management of changing approaches to goals. Chris 
Kennedy's discussion of the relative effectiveness of 
change brought about in a 'top down* manner, where a 
high degree of local commitment is in a sense 'imposed' 
and therefore less of a direct concern of the 
implementing aid agency, as contrasted with that which 
springs from a Tx)ttom up' approach where commitment 
springs naturally from those who will be affected by the 
change, suggested that lasting effectiveness is rarely 
achieved by 'top down' methods on their own and that a 
combination was needed. Both Cyril Weir, in his 
account of an evaluation of a successful project in 
Ecuador, where he used notes and records kept by 
teachers involved in the project to identify changes in 
the way they taught, and Martin Bygate, in his session 
on the development of self-evaluation procedures for 
institutional development in a project with the National 
University of Rwanda, indicated possible lines of 
approach to the development of local involvement and 
commitment. 

Quality versus quantity 

The last of the four important features of education 
projects, and perhaps the most difficult to handle in the 
context of 'value for money' , is the tension between 
quantity and quality which surfaces whenever value is 
discussed in relation to education. The problem is not 
so much one of a conflict between those who demand 
quantity and those who seek quality as one of 
measuring quality in a way which enables it to be 
related to money. Those who are responsible for 
assessing value for money on a ministerial or national 
scale are not typically educationists and the measures 
that they use are those of the economist, that is, they 
are quantifiable and reducible to numerical data. There 
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is, therefore, a real danger that projects in fields such 
as education, which are only partly analysable in terms 
of quantities or numbers, may be assessed on grounds 
that are at best partial and at worst erroneous. 

Danger has, however, a positive aspect in that it 
throws down a challenge, and a successful response to 
that challenge can contribute in a positive way to the 
development of professional perceptions. Let us take as 
an example the problem of measuring change in the way 
teachers teach as a result of a project. The question of 
how best to assess teacher performance has been 
around for a very long time and the answers currently 
offered do not attract general acceptance. Usually 
this is because they rely on a limited range of heavily 
subjective judgements, or because the procedures 
required are impracticable on anything more than an 
experimental basis. Alternatively, they are rejected as 
suspect because there are too many variables between 
teacher performance and the objective data against 
which that performance is set for one to have very 
much confidence in the reliability of the relationship - 
judging teachers solely by the examination results of 
their students is an obvious example of this. 

Here, therefore, lies the challenge - to develop 
instruments for recording and describing what teachers 
do in their class-rooms which produce data which can 
be handled numerically or graphically so that 
meaningful comparisons can be made. 

To do this presupposes as precise a definition as 
possible of the elements of teacher performance so that 
we can be reasonably sure that different observers are 
recording the same thing. It also means that a relatively 
large number of observations need to be made, both 
across a range of teachers and lessons within a short 
period of time, and over a long period of time from the 
start of the project onwards. To ensure the numbers of 
observations required, we need to enlist the teachers 
themselves (or perhaps even their students) as 
observers. This in turn means that the elements of the 
teaching process to be observed should be easily visible 
to the teacher and be such that to record them does not 
appear threatening. 

An example of the kind of instrument that could be used 
in this way was the grid presented by Roger Bowers 
with a range of class-room activity-types arranged in 
relation to four scales: +/- teacher, +/- interaction, 
+/- text, and +/- language. To record the activity- 
types of any particular lesson on a grid of this kind 
would not require much time or analytical ability, while 
a set collected over a long period of time might 
demonstrate a shift in the balance of activity- types 
used by the teacher or teachers. 

Other methods of collecting qualitative data, such as 
the use of diaries, interviews with individuals and 
group discussions, were referred to in the session led 
by Cyril Weir, who stressed that what is important 
about judgemental approaches to evaluation is that they 
'seek to capture and reflect the variety of different 
accounts of events. It is important in this approach to 
capture as full a range as possible of participants' 
differing perspectives rather than to act as if it were 
possible to arrive at only one "correct" view of events. ' 



Conclusion 

The world of education is an extremely complex place 
and a major role of evaluation should be to help our 
understanding of that complexity. It would be 
unfortunate if a narrow definition of what constitutes 
evaluation led to a distorted view of success or failure 
of educational aid projects, with all the implications that 
would have for the funding, management and implemen- 
tation of such projects. It is far from certain that this 
complexity is fully understood by those responsible for 
funding, and it is equally certain that we do not have 
the evaluative mechanisms to demonstrate satisfactorily 
success or progress. 

The time has passed when aid to the education sector 
was considered to be worthwhile in its own right 
without critical examination - the competition for scarce 
resources is too great - and we must learn to survive in 
this new and harsher world. To do so we must learn to 
defend what we do in a way which non -educationists will 
accept. We cannot expect those outside the profession 
to accept any longer our protestations that education is 
different' or to accept our unsupported statements that 
we have achieved something worthwhile. Nor can we 
expect others to develop the tools we need in order to 
demonstrate or support the truth of what we assert. 
This, therefore, is the challenge which was presented 
and discussed at the 1988 Dunford Seminar - to show 
that educational aid projects can be evaluated 
satisfactorily, to develop the instruments with which to 
carry out such evaluations, and to indicate to those who 
exercise financial control the nature of educational 
change, the need to admit a wide range of evaluative 
tools, and the validity of those tools in measuring the 
impact of what we do. 
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The Dunford Seminar: a summary 



The Dunford Seminar, after a brief sojourn at 
Cumberland Lodge in 1987, returned to Dunford House 
in deepest Sussex this year. 

In 1987, the ODA and the English Language and 
Literature Division of the British Council had focused 
the seminar on whether the project formulation (i.e. the 
use of project frameworks and project memoranda) 
employed in other areas of the ODA's work could be 
successfully applied to projects which involved 
education in the broad sense and English Language 
Teaching (ELT) in the narrower sense. 

This year's theme built on the success of the 1987 
seminar with attention now focused on the monitoring 
and evaluation of aid-related ELT projects. 

Twenty-three participants were selected: 15 KELTAs/ 
4 from the British Council's English Language Qualified 
^LQ) cadre, 2 from the private sector (Bell 
Educational Trust and Moray House, Edinburgh), 1 was 
a mathematics TCO and 1 was from the Development 
Division of the ODA in the Caribbean. The decision to 
widen the range of seminar participants came from a 
desire to stimulate discussion and from the current 
interest of the ODA and the British Council in 
collaborating in joint programmes with the private 
sector wherever appropriate. It was unfortunate that 
only two of the five places offered to the private sector 
were taken up but, it is hoped that more institutions 
will be interested in attending in the future. The 
presence of two non-ELT participants helped to situate 
ELT within the ODA's overall country programmes. 

In choosing the theme 'ELT in development aid: 
monitoring aims and measuring results' , the aim was to 
reach a consensus as to how and when monitoring and 
evaluation of ELT projects or the ELT element of larger- 
scale projects should take place and also what form it 
should take. The two-week seminar was made up of the 
following components (see Appendix 1 for the complete 
seminar programme). 

Opening session 

At trie opening session, Brian Vale, Assistant 
Director-General, British Council, and Roger Iredale, 
Chief Education Adviser, ODA, with Peter Freeman, 
Head of Central and Southern Africa Department, ODA, 
and Jack Phillips, Controller, English Language and 
Literature Division, British Council, set the scene for 
the next two weeks by putting the theme of the seminar 
in the context of changes in the management of the 
Technical Co-operation programme and, particularly, of 
the integration of ELT Aid programmes within ODA 
country- wide aid programmes. 

Full-day case studies 

This new component consisted of two full-day case 
studies (India and China), to exemplify how ELT Aid 
projects fitted into the overall aid policies of particular 
countries. These case studies were led by British 
Council staff with the active collaboration, and, in the 
case of India, participation of ODA geographical desk 
officers. 



Visiting speakers 

The role of visiting speakers has been changing since 
the seminars began ten years ago. Initially, leading 
exponents in the ELT field were invited to provide an 
academic stimulus to the seminar by talking about 
recent research they had been involved in. 

When the present seminar was being planned, it was felt 
that, despite the fact that over the past two years the 
theme has been much more sharply focused on ELT in 
development aid, it should still offer academic sessions 
not related directly to the main theme. However, it 
became increasingly clear (and this was confirmed by 
feedback during the seminar) that speakers should, 
wherever possible, give presentations which are 
directly linked to the year's theme. Mick Short's 
session, for example, though enjoyed by the majority of 
the participants, was not felt to have been fully 
appropriate to the theme. 

With other speakers the problem was of trying to slot 
them in at points where they could have maximum effect 
on the case studies. This was generally achieved, 
though feedback indicated that Roger Bowers' session 
would have been more appropriate in the first week. 

To give a flavour of the content of the sessions a short 
report of each follows. They are given in chronological 
order (and if any of them are of particular interest, 
you should contact the speaker through ELLD for 
further information). 



Evaluation and baseline studies 

John Morris 

The speaker explained that, in the ODA, evaluation is 
carried out in order to learn the lesions of experience 
so that benefit can be derived from both the successes 
and failures of projects. It is especially important that 
feedback is given to practitioners so as to increase the 
effectiveness of future programmes. 

Of the eighteen projects a year assessed by the 
evaluation department, very few were ELT projects. 

Various methods of evaluation were explained - 
questionnaires, interviews, unit cost calculations and 
baseline studies. Baseline studies were felt to be of 
great importance if the effectiveness of projects was to 
be judged in quantifiable terms. As a result, projects 
should attempt to draw up ouantifiable evaluation 
procedures, however difficult it was to obtain precise 
data, the startir g-point being a baseline study. 

ELT as a service element in human resource 
development projects: a project manager's view 

Hector Munro 

In this session, the ODA project cycle was explained. 
A project was defined as being determined by 
objectives, and as having financial limits with a budget, 
resources and a time -scale clearly set out. 
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The cycle consisted of: 



Who evaluates? 



• identification of a project 

• preparation (i.e. putting together a project 
memorandum and project framework) 

• appraisal of project memorandum and framework 

• implementation 

• evaluation 

Three types of project were then considered: one where 
ELT was a completely separate element of a larger 
project; one where an ELT project was shoe -horned' 
into an already running project; and one where the ELT 
component was fully integrated at the project design 
stage. The last model was felt to be the best currently 
available and one which would probably increasingly 
replace the KELT programme. 



The language approach to literature and what it 
means for teaching language and literature 

Mick Short 

This session addressed the possibility of using stylistic 
analysis techniques in order to help native and non- 
native readers to understand the process of reading. 
Poems by e e cummings, Stephen Crane and Henry Reed 
were used to provide examples of such techniques, 
which included deviation and parallelism at both the 
semantic and syntactic levels. The possibility of 
using an awareness of these techniques to remedy 
problems of deviance in learner production was also 
discussed. 



Methods of evaluating language programmes 
Cyril Weir 

This presentation looked at the questions which need to 
be addressed when carrying out an evaluation. 

Why evaluate? 

• to understand what is happening in a project 

• to improve a project's chances of success 

• to be accountable 
When to evaluate? 

• at the beginning of a project 

• at stages throughout the life of a project 

• at the end of a project 

• at an interval after the end of a project 



• outsiders 

• insiders 

• both 

How to evaluate? 

• statistical analysis/quantitative data 

• observation 

• interviews 

• group discussion 

The tension between accountability and understanding 
and between quantitative and qualitative evaluation was 
discussed, and it was felt of fundamental importance 
that both aspects needed to be included in any 
programme of evaluation. 



Patterns of control and innovation 

Chris Kennedy 

The speaker argued that evaluation must start at the 
beginning of any project, and that realistic criteria 
should therefore be built in from the outset. In order to 
do this it was important to understand the patterns of 
control and innovation operating within a project. 

Innovation could be either 'top down' or 'bottom up' ; 
as neither extreme precluded the other an interactive 
pattern was therefore proposed. 

Control of administrative, and other, structures could 
either be centralized or decentralized; this would 
depend on the freedom of individual units within the 
system to operate at any level (degree of coupling), the 
extent that any level may adjust any plan or policy 
(degree of plan adaptation) and the role of feedback 
from the lower levels of the system back to the upper 
levels (macro-/micro-perspectives). 

The speaker felt that decentralization could lead to: 

• autonomy 

• commitment 

• efficiency 

• participation 

• relevance 

This would help produce an innovative project, but 
could also result in: 

• coercion 

• conflict 

• dilution of effect 

• disparate outcomes 

• poor co-ordination in planning 

• time wasted 

Although the speaker felt that decentralization was more 
likely to yield positive benefits, he recognized that Lhore 



is in fact little research evidence to suggest that it 
actually works out in this fashion. It was, however, 
important to include this way of looking at projects in 
any project evaluation which is carried out. 

Self evaluation for departmental development: 
aims, prerequisites and approaches 

Martin Bygate 

The speaker argued that ELT evaluation has never been 
a primary concern and when it has been touched on, it 
has tended to concentrate on method effects rather than 
on sub-parts of methods. These efforts have not been 
very successful and the speaker suggested that this 
was because the following points were not taken into 
consideration: 

• evaluation is local, i.e. what works in one place may 
not work in another 

• instead of trying to evaluate whole programmes, it is 
preferable to look for problem areas in specific 
programmes 

• successes and problems in education are intimately 
related to the people involved 

Therefore, if teacher-educators are concerned with the 
well-being of their profession, they must view as a 
priority the development of some element of self- 
evaluation in all teaching units. 

There are also strategic reasons for believing that 
self-evaluation is a useful direction for an aid project to 
take. Firstly, if successful it can enable the 
department involved to be more independent and 
authoritative about its own affairs with all other 
stakeholders. Secondly, the potential scope of the 
project is so wide that there can be some direct interest 
in it for everybody. Thirdly, it is a project whose 
outcomes by definition will remain firmly the property 
of the department. And fourthly, in many respects a 
self-evaluation project promotes the co-ordination of the 
department. 

So, overall, self-evaluation is a crucial dimension 
behind any other innovation. 

ELT projects: a VSO view 
Brian RockclifTe 

This session examined the image that VSO has at large, 
the problems that occur in projects and the role of 
volunteers in solving these problems. The strengths 
and weaknesses of volunteers were outlined as below. 

Strengths: 

• they can extend the impact of a project at a relatively 
low cost 

• they have a strong commitment to work 

• they are likely to be highly motivated to integrate 



• they can be used as non-standard solutions to some 
administrative and logistical problems 

Weaknesses: 

• they are likely to require support in more 
professionally demanding situations 

• they are unlikely to have Third World experience 

• they are a short-term input 

• they have a personal rather than a contractual 
obligation 

• they may lack credibility in some countries. 

Overall, it was felt that VSO volunteers did have a 
potentially important role to play in aid-related projects 
and that consultation with VSO should take place at the 
planning stage of projects so that the volunteers* role 
could be correctly identified and a time-scale drawn up. 



Measuring qualitative change 

Roger Bowers 

This presentation showed how quantitative measure- 
ment could be balanced and supplemented by equally 
valuable qualitative criteria. 

Reporting procedures should be selective, succinct, 
statistical and emphasize stability. But it is also 
important to indicate the long-term progress of a 
project; this can be achieved by producing a clear, 
comprehensive report which provides a commentary on 
the project and emphasizes qualitative rather than 
quantitative change. 

It is also important to address the question of what we 
can and what we cannot change and to be realistic in 
our expectations of the degree to which institutions and 
individuals are likely to change through, or as a result 
of the project. 

Illustrations of different approaches to determining 
effectiveness were provided with reference to the Hong 
Kong Expatriate Teachers Pilot Scheme: 

• the methodological grid for charting class-room 
change 

• defining and monitoring target groups 

• impact on content and impact on process 

• impact on the curricular cycle 

• a combination of these 



ELT and imperialism 

Chris Brum fit 

The underlying theme of this presentation was whether 
exportation of a world-dominant language is 



necessarily malignant. Issues such as those outlined 
below were raised. 

Does the teaching of English throughout the education 
system of a country create dependency because it gives 
rise to unachievable desires and aspirations? If so, 
should English be introduced only at tertiary level? The 
speaker argued that to do so would be to restrict the 
access of the majority of people in a society to the 
corridors of power. 

Are language and culture inextricably linked? Is it 
possible to teach or use the one without reference to the 
burden of the other? The speaker argued that while any 
language may refer to and reflect cultural features, this 
is not an intrinsic feature of language itself. 

It was also argued that the currently accepted view of 
communicative language teaching was deficient in that it 
included only societal dimensions and omitted individual 
and motivational dimensions. 

Finally, it was suggested that the potential for 
imperialism can be reduced through a commitment to 
multi-lingualism and a sensitivity to and support for the 
local indigenous languages and acceptance of the fact 
that some languages other than English also have a role 
to play. 



English language teaching: the role of an 
international publisher in national curriculum 
and materials development projects 

Jeff Andrew 

This session sought to exemplify the advantages of an 
international publisher to the ODA, the British Council 
and to host governments in countries where local 
publishing resources are either inadequate or 
non-existent. 

An international publisher can offer: 

• experienced and skilled authorship 

• professional guidance of editorial staff 

• specialist book design and illustration 

• cost-effective production 

• long-term involvement 

In addition, teacher training was often built in as an 
essential component of any government publishing 
contract by the international publisher. 

A number of successful and not so successful examples 
of ODA, British Council and publisher collaboration 
were given and the session concluded with the speaker's 
belief that international publishers have a crucial role 
to play in the transfer of British skills and professional 
expertise and that ELT textbook projects should be 
planned from the start to involve a publisher. 



The main case studies 

As in previous years the case studies were the most 
important component of the seminar, and they took up 
most of the second week. The aim of the studies was to 
draw up a comprehensive plan for monitoring and 
evaluating the projects in question. Groups of four or 
five participants, by means of briefing documentation 
and discussions with the case study leaders as 
informants, were asked to consider their particular 
project in depth in order to come up with methods for 
evaluating the project. 

The discussions, though formally timetabled, frequently 
spilled over into mealtimes and evenings and the final 
results were presented on the final day. 

The five studies chosen this year (see Appendix 3 for 
details of the monitoring and evaluation procedures 
produced) are given below. 

Cameroon 

This case study considered the joining together of what 
were, up until early 1988, separate projects in the 
secondary sector in the Anglophone provinces of 
Cameroon. One of the projects was a KELT project 
focusing on in-service support for teachers of English 
with a resident KELT Adviser in each of the two 
provinces. The other project aimed to give similar 
support to teachers of mathematics and physics. After a 
review mission by the ODA Education Adviser 
concerned, it was recommended that the projects be 
treated from now on as a single integrated one. 

Case study leaders: Felicity Breet and Francis 
Lopez-Real 

Oman 

This case study dealt with a curriculum -development 
and teacher-education project six months into its 
implementation and discussed the proposed monitoring 
and evaluation strategies. 

Case study leader: Ian Harrison 

Zambia 

This case study dealt with the aims of teaching English 
at primary level in the light of the educational reform 
document of the Ministry of General Education and 
Culture. 

Case study leaders: Leo O'Keefe and Peta Constable 
Ethiopia 

This case study dealt with the role of ELT at university 
level with respect to its overall effectiveness in the 
development of Ethiopian education. 

Case study leader: Rod Hicks 

Indonesia 



^ ffris case study considered the role of the ELT Projects 
I Unit which co-ordinates ODA ELT projects in 
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universities in Indonesia (this unit adopts a different 
approach to project implementation to the standard' 
project framework). 

OEAD/ELMDO work 

In addition, there were sessions on the work of 
Overseas Educational Appointments Department 
(OEAD) and English Language Management 
Department Overseas (ELMDO). 

The theme of OEAD's session was the way projectisation 
is affecting the recruitment and training of KELTAs. It 
was felt strongly that management training was needed 
in the form of short practical courses rather than 
longer-term academically oriented management courses. 

The ELMDO session explained how the structure and 
activities of ELMDO have been reorganized since April 
1988 and the way in which the new ELT Aid Unit would 
function and serve the professional needs of all KELTAs 
by working on the procedures for monitoring, 
management, the production of reports, briefing, 
recruitment and supply. 

Extra tasks 

There were three additional tasks for the case study 
groups: 

• to suggest a new format for the part of the present 
KELT post report dealing with professional matters. 
These suggestions would be fed into the preparation 
of new reporting procedures which will be more in line 
with the present project formats 

• to examine and suggest amendments and additions to a 
classification scheme for ELT Aid projects which will 
be of use in the better storage and handling of 
information on ELT projects around the world (see 
Appendix 4) 

• to examine and comment on the draft project summary 
form which was intended both as a day-to- day 
management check -list and as a ready reference 
containing basic non-confidential information which 
KELTAs and other interested parties can consult (see 
Appendix 4) 

Evaluation 

A seminar on the theme monitoring and measuring 
results' obviously had to include its own dose of self- 
evaluation! This took the form of a mid-seminar and 
end-of-seminar written evaluation (see Appendix 5 for 
copies of the evaluation form). In addition, informal 
feedback sessions were held at the end of the first week 
and on the final morning, on this occasion with the 
presence of Roger Bowers, Deputy Controller, ELLD, 
British Council and Roger Iredale of the ODA. 



Appendix 1 
Seminar programme 



18-29 July 


0900 to 1100 


1115 to 1300 


1400 to 1600 


1615 to 1800 


Monday 
18 








Course participants 
arrive 



Tuesday 
19 



Admin 

John Hilton (BC) 
Nick Butler (BC) 
Opening session 
Peter Freeman 
(ODA) 

Brian Vale (BC) 
Roger Iredale (ODA) 



Round table 
Peter Freeman 
(ODA) 

Brian Vale (BC) 
Roger Iredale 
(ODA) 

Jack Phillips (BC) 



Plenary 

John Morris (ODA) 
Plenary 

Hector Munro (BC) 



cont. 

Hector Munro 



Wednesday 
20 



Plenary 
Mick Short 
(Lancaster Univ.) 



China Case 
Jaspar Utley (BC) Mick Short 
Stephen Magee (Lancaster Univ.) 

(ELT lecturer) 



Study 

Jack Phillips (BC) 



Thursday 
21 



Plenary 
Cyril Weir 
(Reading Univ.) 



Helen Boyle (BC) 
Kate Mulvey 
(ELT consultant) 



India Case 

Pam Scutt (ODA) 



Study 

Simon Cole (BC) 
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Friday 
22 



Plenary 
Chris Kennedy 
(Centre for British 
Teachers) 



Plenary 
Martin Bygate 
(Reading Univ.) 



Introduction of 
for group work 

Felicity Breet 

(KELT) 

Pete Constable 
(KELT) 



case studies 

in week 2 

Rod Hicks (KELT) 

Ian Harrison (KELT) 

Geoffrey Crewes 

(KELT) 



Saturday 
23 


Case studies 


Case studies 
Seminar feedback 






Monday 
25 


Plenary 
Brian Rockliffe 
(VSO) 


Case studies 


Case studies 


Case studies 


Tuesday 
26 


Plenary 
Roger Bowers 
(BC) 


Case studies 


OEAD/ELLD 
Brian Smith (BC) 
Tony Deyes 
(ELT Aid Unit) 


cont. 


Wednesday 
27 


Plenary 
Chris Brumfit 
(Southampton 
Univ.) 


Report structure 
and format 
Tony Deyes 
(ELT Aid Unit) 


Case studies 


Case studies 


Thursday 
28 


Plenary 
Jeff Andrew 
(Longman 
International) 


Case studies 


Case study 
reports 


Case study 
reports 


Friday 
29 


Course round-up and evaluation 
John Hilton (BC) Roger Bowers (BC) 
Nick Butler (BC) Roger Iredale(ODA) 


Course participants 
depart 
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Appendix 2 

List of participants 



KELT Advisers/TCOs 

Felicity Breet 
Francis Lopez-Real 
Stephen Magee 
David Bonamy 
Rod Hicks 
Jane Carey 
Geoffrey Crewes 
Mark Todd 
Mike Connelly 
Ian Harrison 
Mark Roberts 
Eddie Uprichard 
Patrick McLaughlin 
Roger Barnard 
Pete Constable 
Leo OTCeefe 



Cameroon 

Cameroon 

China 

Egypt 

Ethiopia 

Guinea Conakry 

Indonesia 

Mozambique 

Nepal 

Oman 

Pakistan 

Togo 

Turkey 

Yemen A R 

Zambia 

Zambia 



ODA staff 

Mike Ratcliffe 

British Council staff 

Neii Gilroy-Scott 
Huw Williams 
David Clarke 
Gordon Slaven 



India (des) 
Kenya 
ELMEKO) 
ELMEKO) 



Outside institutions 

Lewis Kerr 
Les Dickinson 



Bell Educational Trust 
Moray House 



Seminar staff 

John Hilton 
Nick Butler 
Jay Mehta 



Seminar director 
Assistant director 
Course assistant 



Appendix 3 

Monitoring and evaluation procedures: case studies 



Draft exercise 



Cameroon case study 

Illuminative evaluation - some suggestions 
Indicators 



Impact on 
target group 

1 Students 



2 Other teachers 



3 School authorities 



4 Education 
department 



5 Other ELT 
professionals 



6 Public opinion 

7 Aid agency 



1.1 Increased student talk about Maths/Physics 

1.2 Increased use of M/P library books 

1.3 Demand for science club activities 

1.4 Dissatisfaction with too much testing 

2.1 Inclusion of M/P contexts in lessons 

2.2 Disruptive effect of M/P/E project activities on time-tables, 
lessons, etc. 

2.3 Increased courses with language in lessons 

2.4 Increased/decreased status from project/non-project staff 

3.1 Increased demand for school library/books 

3.2 Increased administration burden on Principal 

3.3 Other teachers unhappy at additional M/P/E resources 

3.4 Disruptive effect of teachers' release from seminars 

4.1 Increased administration burden - circulars, reproduction 

4.2 Disruption of routine of staff 

4.3 Increase in report writing/meetings 

4.4 Increased/decreased status for staff involved/not involved 

4.5 Distortion of staff relations/management lines 

5.1 Increased/decreased status associated with M/P project 

5.2 Certainty/lack of professional capacity - integrated elements 

5.3 Conflict of line management/role : integrated versus specialism 

5.4 Increased/decreased value of integrated element: professional 
development 

6.1 Survey of public awareness of project 

6.2 Students reading/talking more at home: M/P : survey 

7.1 Role/time conflict on Representative/British Council staff 

7.2 Round of project-related presentations/openings/discussions 

7.3 Increased demand on ODA desk officer 



Source 



Class teacher 
Teacher/students 
Teacher 
Teacher 

Principal/Assoc. 

Principal 
English Assoc. 
Principal 

Principal 
Principal 
Principal 
Principal 

Inspectors 

Delegate/Insp. 

Delegate/Insp. 

Delegate/Insp. 

Delegate/Insp. 

Adviser 
Adviser 
Adviser 

Adviser 

Adviser/Insp. 
Principal 

Representative 
Representative 
ODA 



20 



Draft exercise 



Ethiopia case study 

1 Presentation of case study results 

The overall aims of the project and the four strands of 
its output, i.e. Service English Language Improvement 
for teachers, a TEFL course for teachers and the post- 
graduate courses were outlined. 

2 Monitoring 

The case study presented an approach to monitoring 
using the spoken English course to exemplify this 
approach - an approach which would be used for all 
components - using specified instruments and 
procedures. It then presented an approach to overall 
evaluation of the project through specific instruments 
applied through a baseline study and a summative 
evaluation. 

Objectives 

• to ensure quality of course design 

• to ensure quantity control of courses produced, etc. 

• to ensure quality control of implementation 

• to check that project objectives are achieved 
according to schedule 

• to ensure that monitoring continues after the life of 
the project 

• to use monitoring procedures in the training of MA 
(TEFL) and Ph.D. students 

Procedures 

Multi-method approach: data obtained from 

• questionnaires - teachers and students 

• exam results 

• observation by KELTAs/MA and Ph.D. students 

• discussions with teachers and students 

• PF/memorandum/project output chart 

Personnel 

• university teachers 

• university students 

• KELTAteam 

• external advisers 

Timing 

• at key points in the life of the project, e.g. end of 
trialing, committee stages, etc. 

• continuous monitoring of outputs against project time 
frame 



Reporting 

• RH annual report/project report 

• jet-in, jet-out reports 

Primary users 

• internal AAU - course writers; English Dept; 
MA and Ph.D. students 

• external, ODA, British Council 

Cost 

• jet-in, jet-out visits 

• paper, files, etc. 

• database program (+ instructions!), disks 

• time (difficult to quantify) 
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Draft exercise 



Monitoring flow-chart 



teacher feedback/ 
workshop 



questionnaire for 
students 



z 



NEEDS ANALYSIS 



X 



observation 



} 



SYLLABUS DESIGN 





Draft exercise 



Monitoring procedure 



Immediate objectives 


Indicators of achievements 


Instruments 


Action/ 
Resp 


Dates 


Introduce new spoken 
English course for B.Ed. 
students 


course/materials in use 


project report 


RH 


07/91 


1 Needs analysis 


needs profile established 


past & present student quest 
discussions to literature and 
methodology teachers 


CWrs 


10/87 


2 Design syllabus 


committee approval 


minutes/reports 


English 
dept 


12/87 


3 Prepare materials 


materials in class 


check 


RH 


12/87 





4 Trial and rewrite 



revised material in use 



external evaluation 
report class 
observation 
reports teacher 
reports/questionnaire 
exam results 



outsider 
MA students 
CWrs 
Trs 



01/88 



07/89 



5 Appraise 



committee approval 



minutes/report 



Research 
and 

publications 
committee 



07/88 



6 Produce 



materials printed 
materials recorded 



check 



Trs 



10/89 



7 Evaluate 



report 



class 

observation 
reports 

teacher reports/ 
questionnaire 
student discussions 
exam results 



RH 


Rod Hicks 


CWrs 


Course Writers 


Dept 


English Dept 


Outsiders 


Visiting 




consultant (UK) 


Trs 


Teachers 



review/report 



MA students 
CWrs 



CWrs 
CWrs 
Trs 



07/90 



02/91 



percentage of students from student 
spoken English course records 
entering TEFL methodology 
course 



10/91 



FRIC 



Co 



3 Overall evaluation 



The following procedure was agreed upon for evaluating 
the overall impact of the project and judging to what 
extent English had become more effective as a medium 
of instruction in secondary and post-secondary schools. 
There would be a baseline study carried out in October/ 
December 1988 in the university and senior secondary 
schools involving three specific instruments for 
quantitative assessment and three tools for gathering 
and collating descriptive and qualitative indicators. 

The same instruments and indicators would be applied 
to a cross-section of university students and lecturers 
and to senior secondary school teachers and students 
during the baseline study and the summative evaluation. 

• The baseline study October/December 1988 applied to 
secondary school 10/11 grade. 

• The baseline study October/December 1988 applied to 
university students. 

• The baseline study in third/fourth year of medical, 
science, geography and history faculty. 

• The summative evaluation October/December 1992 
for the same years and faculties in the university. 

• The summative evaluation April/June 1996 for 
secondary schools (as only then would the effects of 
the new 'improved' language teachers feed through 
from the university course to effect the subject 
teaching in the schools). 

NB 

1 A further evaluation would in the ideal world be 
carried out in 1999 to judge the full effects of 
improved teacher training. 

2 A tracer of B.Ed, students leaving university in 
1992/3 will have to be made so that evaluation can be 
focused on those r 'hools receiving the newly trained 
teachers. 



Draft exercise 



Indonesia case study 
ELT projects unit 

Wider objective: To improve the effectiveness of 
overseas training offered by aid programmes to 
facilitate the transfer of technology. 

Indicated by: A decrease in the number of overseas 
training opportunities not taken up as a result of low 
proficiency in English. 

Immediate objective: To develop a sustainable 
functioning unit that can provide relevant and effective 
ELT for postgraduate students. 

Indicated by: An improved and recognized provision of 
appropriate and institutionally co-ordinated English 
language instruction. 

Although all projects are different in that each host 
institution is different, they all have as their aim 
institutional development, i.e. concentrating on the 
local institution itself and trying to get it to do what it 
is supposed to do more effectively. 

Evaluation chart 

(see also following pages) 



START 

Sept 

1988 



(Oct 
1988) 



Externa] 
evaluation 



1 


2 


3 


Implement 


END 








4 


19 


Staff 


Needs 


Plan 


Evaluate (internally) 
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4.1 




Syllabus 





+1 



4.2 




Materials 





4.3 




Testing 





4.4 
Resources 
management 



* 



4.5 
Train ig 
development 



4.6 




Institutional 




structure 





4.7 




Acceptance 





K 



N 



Draft exercise 



Output 


Indicator 


How assessed 


1.1 Director of studies (DOS) or 
counterparts identified 

1.2 Teacher/supervisor (T/S) 
recruited 


in place 
in place 


noted by 

• Co-ordinator of projects unit 
(C/PU) 

• Dean 


2. 1 Identification of institution's 
needs and priorities 


report by T/S 


approved/assessed by 

• C/PU 

• Dean 


3.1 Formulation of plan to 

implement identified priorities 


produced by T/S + C/PU 


approved/assessed by 

• C/PU 

• Dean 


4. Plan implementation 






4.1 Syllabus 






4.1.1 process 


system in place for continued 
syllabus revision 


* syllabus files, master 

• minutes of co-ordination 
meetings 


4.1.2 product 


completed syllabus in line with 
objectives 


• syllabus files 


4.2 Materials 






4.2.1 process 


regularly improved bank of trialled 
materials 


• minutes 

• teachers' reports 


4.2.2 product 


completed materials in line with 
syllabus 


• student questionnaires 

• teachers' reports 

• see library, files, catalogues 


4.3 Testing 






4.3.1 process 


system of regular test revision and 
development 


• see test file 

• test revision manual 

• test specifications 


4.3.2 product 


bank of pre/progress/post-tests and 
items in line with objectives and 
syllabus 


• see test file 


4.4 Management of learning 
resources 


secure but accessible systems 
trained resource supervisor 


• catalogue system 

• maintenance schedule 

• report by T/S & C/PU 
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Draft exercise 



Output 



Indicator 



How assessed 



4.5 Training development 
4.5.1 counterpart 



carrying out functions and 
responsibilities in line with job 
specification 



reports by 

• T/S 

• C/PU 

• counterpart 



4.5.2 teachers 



improved performance in accordance 
with teacher profile 



• teacher profile (beginning/ 
interim/at end) 

• reports by T/S and DOS of 
teacher's performance in 
workshops 

• student questionnaires 

• test results (students) 



4.6 Institutional structure 



premises 

established post for DOS 
capital investment 
recurrent funding 



sight/assurance/promise from wider 
institutions 



4.7 Acceptance/integration 
4.7.1 with wider institutions 



2 -way communication flow 

SEU client departments 

SEU posts part of institution's 
career structure 

continued institutional support and 
use of SEU 



percentage age of course 
participants nominated for exit 
tests 

micro-political snooping 



• course files 

• minutes of meetings 
SEU <r* client departments 



4.7.2 with national training 
programme 



use by external institutions in 
line with national programme 



• reports by external agencies 

• number of students sent by 
external agencies 
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Oman case study 



Curriculum renewal project: 
student materials evaluation 

A questionnaire will be printed in the Teacher's Book at 
appropriate intervals - probably after every Unit which 
represents a week's class-room work. All teachers will 
thus be completing the questionnaire, most of the 
responses to which can be quantified. Staff who do not 
necessarily teach materials but who are asked for 
comments can also use the same questionnaire. 

A further small set of questions will be required for 
evaluation of the introduction to the Teacher's Book. 

Yet another (observation) questionnaire will probably be 
required for use by personnel who are asked to observe 
selected teachers using the materials where the emphasis 
will be on the materials rather than on the teacher. 

Teachers will have to be sensitized/won over to 
completing the questionnaire, probably through meetings 
and/or training courses. It is also possible that further 
meetings may be required in order to clarify certain 
responses to certain parts of the questionnaire. 

Consideration must also be given to using the question- 
naires in different regions in different ways in order to 
try and reduce the load for individual teachers. One 
idea might be for regions A and B to provide information 
on Units 1, 6, 10, etc. while regions C and D would 
provide information on Units 2, 7, 11, etc. 

The schedule allows for some piloting of the question- 
naire to take place in 1988/89, the year before the 
materials are due to be trialled. This piloting is seen to 
be a necessary part of the development of the 
instrument. 

The training aspect of the questionnaire is one that 
should not be forgotten, in that by getting teachers to 
think critically about materials - along certain set lines - 
one is alsc hopefully achieving some teacher 
development. 

Further consideration perhaps still needs to be given to 
finding ways of focusing the questionnaire more on 'new 
departures' . 

Some of the questions are designed to be completed only 
by non-teachers - these may be Regional Inspectors, 
Curriculum and Training Officers (CATO), etc. No 
indication of which questions these are has been given. 



Draft exercise 
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Improvement of inspectors' performance 

1 Initial performance of inspectors (1988-1989) 

a familiarization with Curriculum Renewal Project 
(CRP) materials and methodology 

b acceptance of above 

c ability to use INSET materials for CRP 

Indicators of assessment 

a percentage attendance of inspectors at familiariz- 
ation course at which (specified) CRP materials are 
shown and explained 

b questionnaire of inspectors' attitudes to certain types 
of material and methods, issued before (?) familiariz- 
ation course, checked again at the end of the course 

c report by CATOs of inspectors' ability to prepare and 
peer- teach INSET materials during familiariz- 
ation course 

2 Performance of inspectors as from September 1989 

a to ensure that teachers are using CRP materials in 
appropriate ways 

b to advise teachers how to make best use of CRP 
materials 

c to advise Ministry of Education of the implementa- 
tion of CRP materials in schools (i.e. in what specific 
ways the T/Ss are applying CRP, well or badly) 

Indicators of assessment 

a inspectors' reports of lesson-observation sent to 
Ministry of Education and CATOs' reports of joint 
(inspector/CATO) lesson-observations 

b reports of post-lesson discussion with T/S sent to 
Ministry of Education; and CATOs' reports of joint 
(inspector/CATO) discussion with T/S and 
inspectors' reports of INSET meetings sent to 
Ministry of Education; (part of) and T/S's question- 
naires on INSET meetings sent to Ministry of 
Education CATOs' reports of joint (inspector/CATO) 
INSET meetings 

c inspectors' report of class-room use of CRP sent to 
Ministry of Education; and copies of T/S's tests and 
exams (annotated by inspectors) sent to Ministry of 
Education 

Improvement of teachers' performance 

Indicators of assessment 

a baseline behavioural study of non-CRP teachers by 
inspectors using same check-list below, to be used for 
a "state of play 1988-89' statement to be evaluated in 
1993 

b baseline measurement of student performance 1988 
( culture-free examinations' ) to be evaluated in 1993. 
Quantitative 

c baseline perception of teachers' classroom 
performance comprising a list of 'n' teacher- 
weaknesses, based on data from inspectors 



d teacher self-evaluation questionnaire completed at the 
end of initial Inset courses 

e inspector reports based on observation of CRP 
teachers 

f Ministry of Education/CATO reports based on 
observations of teachers in schools 

g external (ODA) reports based on 5, 6 (see section 
below) and their own visits 

Improvement of teacher and inspector performance 

Output : Appropriate guidance materials for teachers and 
inspectors 

1 Code of practice for school visits by inspectors/ 
trainers, CATOs, etc. drafted before the 
familiarization course. This code to be modified and 
agreed upon during the familiarization course. 

2 Questionnaire issued to inspectors before 
familiarization course, to: 

a elicit their attitudes towards appropriate ELT 
techniques 

b elicit impressions of 'typical' teacher behaviour; 
these would inform a pre-CRP baseline study for 
evaluation in 1993 

c elicit (ten) general weaknesses among teachers 

3 INSET packs (print/audio/video) dealing with (n) 
CRP-related topics, for use by inspectors/CATOs in 
INSET meetings. (Some of) these packs to be trialled 
in familiarization course through peer -teaching. 

4 Programme devised for familiarization course for 
inspectors. 

5 Pro forma class-room observation reports devised: 

part 1: observation check-list, including the 
weaknesses derived from 2c above; inspector would 
indicate T ability on a (1-4) scale 
part 2: notes on practicability of CRP materials 
used in class 

part 3: record of points discussed with T afler 
lesson 

part 4: samples of most recent monthly test and/or 
mid-year exam produced by th. teacher, together 
with comments by the inspector 

6 Questionnaire devised for teachers at the end of the 
series of INSET meetings to elicit self-evaluation 
judgements and to include some questions designed to 
assess the effectiveness of the inspector as trainer. 

7 Questionnaire/report-form devised for inspectors 
after a series of INSET meetings. 

8 Materials evaluation questionnaire devised for 
teachers after they have applied (one) unit of CRP 
materials. 

9 'Context free' attainment tests devised for students 
to take in March 1989 and in March 1991, 1992, 1993; 
these will form baseline study of student attainment 
pre-CRP, and subsequent evaluation. 
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Draft exercise 



Zambia case stud y 

1 Monitoring and evaluation procedures for ODA purposes 
What? 

Year one 

• Baseline study to provide profile of teacher class-room 
behaviour grades one to four. This is based on a 
check-list of required behaviour to use new course 

• Baseline study of pupil performance at end of grades 
one to four 

Year two 

• Baseline study to provide profile of teacher class-room 
behaviour grades five to seven 



• Baseline study of pupil performance at end of grades 
five to seven 

• Book One in schools: monitor distribution to districts, 
schools and pupils 

Year three 

• Book Two in schools: monitor distribution to districts, 
schools and pupils 

Book One: assess state and extent of use in the schools 

• Evaluation of teacher class-room behaviour grade one, 
after two years of new course 



• End-of-year tests for grade one pupils in baseline 
schools 

Year four and one 



By whom? 



UK consultant + 
Zambian team 



ditto 



UK consultant + 
Zambian team 



ditto 



Inspectorate/ 
FINN IDA/CDC 



Inspectorate/ 
FINN IDA/CDC 



UK consultant + 
Zambian team 



administered by 
inspectors 



As for year three, moving up a grade each year. At the end of year nine, a summative report 
on the entire course can be written, including: 



• Performance in grade 7 examination 

• Improved occupational appropriacy 

• Improved pupil preparation for grade 8 



Note 

CDC Curriculum Development Council 
FINNIDA Finnish aid agency 



Exam Board 

Evaluation Dept, 
CDC 



G8 teachers (all 
subjects) 



How? 



profiling following 

class-room 

observation 

attainment tests 



profiling following 

class-room 

observation 

attainment tests 



observation/ 
receipt form 



observation/ 
receipt form 



profiling following 

class-room 

observation 

attainment tests (as 
year one) 



statistics 

survey of employers, 
baseline year seven, 
biennially 

reports 
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Draft exercise 



2 Monitoring and evaluation procedures for internal purposes 
WLat? 

Year one 

Monitor the establishment of: 

New materials' awareness campaign through 

• quarterly (?) newsletter 

• radio broadcasts 

INSET programme for: 

• inspectors 

• resource teachers 

• grade one teachers 

Check that resource centres are functioning 
Monitor 

• writing of the course 

• piloting procedures 

• the revision process 

• the printing and distribution 

Year two 

Evaluate effectiveness of awareness campaign 



Evaluate INSET programme for grade one teachers, year one 



By whom? 



project co-ord ; nator 



ditto 



inspectors/project 
co-ordinator 

ZEMP/PINNIDA 



CDC/inspectors 
co-ordinator 



inspectors/resource 
teachers 



Initiate and monitor support programme for teachers using the new ditto 
book 

Evaluation of acceptability of grade one material to teachers, pupils, senior inspectors 

parents and inspectors primary 



Monitor 

• writing of the course 

• piloting procedures 

• the revision process 

• the printing and distribution 

Year three 

As for year two, moving up a grade each year. At the end of year 
nine, a summative report on the entire course can be written. 



ZEMP/PINNIDA 



How? 



informal meetings/ 
problem-solving 



ditto 



visits/reports 



supervision adviser's 
progress reports 



reports/feedback 
from interested 
parties 

observation/ 
feedback 
from teachers 

regular meetings 



country- wide 
informal contacts 

supervision adviser's 
progress reports 
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Appendix 4 

ELT projects: a classification scheme 



With a view to keeping KELT Project Summaries on disk 
for easy reference and updating, it would seem useful if 
we could give each project a code classifying it according 
to a number of criteria. If the records are kept in a 
database format it would enable us to 'pull out' all 
projects of a particular kind easily and thus answer 
questions such as "how many in-service teacher training 
projects at primary level are there?" It would also enable 
KELTs thinking ahead to their next posting to look at 
the range of projects running which might fit their own 
particular interests or expertise. 

I give below some initial thoughts on a possible three- 
group scheme of classification and would welcome any 
comments, additions, modifications, etc. 

Each project would be allocated to one of the options 
at each group and given the appropriate coding. The 
end result would be rather like a file reference with 
three three-letter codes separated by slashes. 

Group I: Situation 

1 English as a foreign language EFL 

2 English as a second language ESL 

3 English as a mother tongue EMT 

Group II: Level 



1 

2 
3 
4 
5 
6 



Primary 
Secondary 
Tertiary 
Non-formal 



Distance 
General 



PRI 

SEC 

TER 

NFL 

DIS 

GEN 



Group III: Type 



1 

2 
3 
4 
5 
6 
7 



General purpose English 

General ESP 

EAP 

EOP 

Service English 
Materials development 
Testing, evaluation and 
examinations 

Pre-service teacher training 
In-service teacher training 
Inspectorate 

General advisory at Ministry 
level 



GPE 
ESP 
EAP 
EOP 
SER 
MAT 



8 
9 
10 
11 



TES 
PST 
1ST 
INS 



MIN 
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Appendix 5 

Seminar evaluation - Part One 



Please set down, in order of importance if possible, the 
points you would like raised at the final round-up 
session on Friday morning under the main headings that 
follow. Your points can be criticisms of the way this 
year's seminar has been handled or positive suggestions 
for future seminars. We will aim to structure the final 
session so as to give a reasonable amount of time for 
discussion and comment on each area. 

1 This year's theme: interest, relevance and value 



2 The structure and organization of the seminar 



3 The choice of main speakers: interest, relevance and 
value 



4 The administration of the seminar 



5 Suggested themes for next year's or future seminars 



6 Any other comments, criticisms or suggestions 



In order for the chair of the final session to select, at 
least initially, one or more people to start the ball rolling 
under each heading at the final session, it would be 
helpful if you could give your name on this sheet, 
though of course you are not obliged to do so. Since you 
will have completed this sheet before some of the seminar 
activities have taken place, you will be given a short 
questionnaire to complete at the end of the final session 
to cover the seminar as a whole. 



(1) 



(2) 



(3) 



Name 
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Seminar evaluation - Part Two 



Please give a rating for each session on a scale 1 to 5 
(bad to excellent) under each heading. 

1 Opening session 

Relevance [^Quality [^Interest [^Overall rating | | 
Comments: 

2 John Morris - Plenary 

Relevance [^Quality [^Interest [^Overall rating | | 
Comments: 

3 Hector Munro - Plenary 

Relevance [^Quality | [interest | [Overall rating | | 
Comments: 

4 Mick Short - Plenary 

Relevance [^Quality [^Interest [^Overall rating | | 
Comments: 

5 China case study 

Relevance QQuality I [interest [^Overall rating I I 
Comments: 

6 Cyril Weir - Plenary 

Relevance [^Quality dJlnterest [^Overall rating 
Comments: 

7 India case study 

Relevance QQuality Olnterest [[^Overall rating I I 
Comments: 

8 Chris Kennedy - Plenary 

Relevance [^Quality Olnterest QOverall rating I I 
Comments: 

9 Martin Bygate - Plenary 

Relevance OQuality Qlnterest OOverall rating I I 
Comments: 



10 Case study previews 

Relevance Q Quality Q Interest Q Overall rating | [ 
Comments: 



11 Brian Rockliffe - Plenary 

Relevance Q Quality Q Interest Overall rating | | 
Comments: 

12 Roger Bowers - Plenary 



Comments: 

13 OEAD/ELLD session 

Relevance Q Quality Q Interest Q Overall rating [ | 
Comments: 

14 Chris Brum fit - Plenary 

Relevance Q Quality Q Interest Q Overall rating | | 
Comments: 

15 Report format session 

Relevance [^} Quality Q Interest Overall rating I I 
Comments: 

16 Jeff Andrew - Plenary 

Relevance O Quality CZ| Interest O Overall rating I I 
Comments: 

17 Case studj report-back session 

Relevance Q Quality Q Interest Q Overall rating I I 
Comments: 

18 Final round-up session 

Relevance Q Quality Q Interest Q Overall rating | | 
Comments: 



Relevance 



I | Quality Q Interest Q Overall rating | | 
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19 The case study exercise as a whole 

Relevance! | Quality Q Interest Q Overall rating | | 

Comments: 



20 The seminar as a whole 

Relevance Q Quality Q Interest Q Overall rating! | 
Comments: 

21 Any other comments 



